ARCHITECTURAL HERITAGE 3D MODELLING USING UNMANNED AERIAL VEHICLES MULTI-VIEW IMAGING

Today, Architectural Heritage 3D models are created using Unmanned Aerial Vehicles (UAV) imagery and processing through Computer Vision (CV) methods. They are becoming more acceptable as reliable sources for study, documentation, diagnostics, intervention planning, monitoring, and management decision-making. The Deir-e-Kaj caravanserai, located in Qom, Iran, is a massive and half-destroyed architectural heritage that belongs to the Seljuk era. The obstructed access due to illegal deep excavations and the extensive demolished structure did not allow for a complete mapping using traditional and terrestrial techniques. Considering the condition and vulnerability of the artifact, it looks necessary to use a safe, non-destructive, and remote method for 3D documenting. The literature review shows in most of the research UAVs are used for acquiring nadir images, which are combined with the terrestrially acquired data for complete 3D modelling. In this case, a multi-view UAV imaging strategy is considered for the as-is 3D modelling of Deire-e-Kaj. Three different imaging angles are designed and used to carry out the comprehensive and all-needed data acquisition. The nadir images are acquired to cover the plan and enclosure, and the horizontal and oblique images cover the façades and interior spaces of the artifact. Adopting a suitable photogrammetric process based on the SfM workflow allows for obtaining an accurate, high-quality, and textured 3D model of the caravanserai. Accuracy evaluation of the result using Ground Control Points shows a total accuracy of ±1 cm. This study demonstrates the efficiency of multi-view UAV photogrammetry as a rapid, safe, and precise method to create a complete 3D model of massive, hard-to-access, and vulnerable Architectural Heritage

EFFECT OF KEYFRAMES EXTRACTION FROM THERMAL INFRARED VIDEO STREAM TO GENERATE DENSE POINT CLOUD OF THE BUILDING'S FACADE

Keyframes extraction is required and effective for the 3D reconstruction of objects from a thermal video sequence to increase geometric accuracy, reduce the volume of aerial triangulation calculations, and generate the dense point cloud. The primary goal and focus of this paper are to assess the effect of keyframes extraction from the thermal infrared video sequence on the geometric accuracy of the dense point cloud generated. The method of keyframes extraction of thermal infrared video presented in this paper consists of three basic steps. (A) The ability to identify and remove blur frames from non-blur frames in a sequence of recorded frames. (B) The ability to apply the standard baseline condition between sequence frames to establish the overlap condition and prevent the creation of degeneracy conditions. (C) Evaluating degeneracy conditions and keyframes extraction using Geometric Robust Information Criteria (GRIC). The performance evaluation criteria for keyframes extraction in the generation of the thermal infrared dense point cloud in this paper are to assess the increase in density of the generated three-dimensional point cloud and reduce reprojection error. Based on the results and assessments presented in this paper, using keyframes increases the density of the thermal infrared dense point cloud by about 0.03% to 0.10% of points per square meter. It reduces the reprojection error by about 0.005% of pixels (2 times)

DECISION-BASED FUSION OF PANSHARPENED VHR SATELLITE IMAGES USING TWO-LEVEL ROLLING SELF-GUIDANCE FILTERING AND EDGE INFORMATION

Pan-sharpening (PS) fuses low-resolution multispectral (LR MS) images with high-resolution panchromatic (HR PAN) bands to produce HR MS data. Current PS methods either better maintain the spectral information of MS images, or better transfer the PAN spatial details to the MS bands. In this study, we propose a decision-based fusion method that integrates two basic pan-sharpened very-high-resolution (VHR) satellite imageries taking advantage of both images simultaneously. It uses two-level rolling self-guidance filtering (RSGF) and Canny edge detection. The method is tested on Worldview (WV)-2 and WV-4 VHR satellite images on the San Fransisco and New York areas, using four PS algorithms. Results indicate that the proposed method increased the overall spectral-spatial quality of the base pan-sharpened images by 7.2% and 9.8% for the San Fransisco and New York areas, respectively. Our method therefore effectively addresses decision-level fusion of different base pan-sharpened images

AUTOMATIC ROAD CRACK RECOGNITION BASED ON DEEP LEARNING NETWORKS FROM UAV IMAGERY

Roads are one of the essential transportation infrastructures that get damaged over time and affect economic development and social activities. Therefore, accurate and rapid recognition of road damage such as cracks is necessary to prevent further damage and repair it in time. The traditional methods for recognizing cracks are using survey vehicles equipped with various sensors, visual inspection of the road surface, and recognition algorithms in image processing. However, performing recognition operations using these methods is associated with high costs and low accuracy and speed. In recent years, the use of deep learning networks in object recognition and visual applications has increased, and these networks have become a suitable alternative to traditional methods. In this paper, the YOLOv4 deep learning network is used to recognize four types of cracks transverse, longitudinal, alligator, and oblique cracks utilizing a set of 2000 RGB visible images. The proposed network with multiple convolutional layers extracts accurate semantic feature maps from input images and classifies road cracks into four classes. This network performs the recognition process with an error of 1% in the training phase and 77% F1-Score, 80% precision, 80% mean average precision (mAP), 77% recall, and 81% intersection over union (IoU) in the testing phase. These results demonstrate the acceptable accuracy and appropriate performance of the model in road crack recognition

CMGFNet: A deep cross-modal gated fusion network for building extraction from very high-resolution remote sensing images

The extraction of urban structures such as buildings from very high-resolution (VHR) remote sensing imagery has improved dramatically, thanks to recent developments in deep multimodal fusion models. However, Due to the variety of colour intensities with complex textures of building objects in VHR images and the low quality of the digital surface model (DSM), it is challenging to develop the optimal cross-modal fusion network that takes advantage of these two modalities. This research presents an end-to-end cross-modal gated fusion network (CMGFNet) for extracting building footprints from VHR remote sensing images and DSMs data. The CMGFNet extracts multi-level features from RGB and DSM data by using two separate encoders. We offer two methods for fusing features in two modalities: Cross-modal and multi-level feature fusion. For cross-modal feature fusion, a gated fusion module (GFM) is proposed to combine two modalities efficiently. The multi-level feature fusion fuses the high-level features from deep layers with shallower low-level features through a top-down strategy. Furthermore, a residual-like depth-wise separable convolution (R-DSC) is introduced to enhance the performance of the up-sampling process and decrease the parameters and time complexity in the decoder section. Experimental results from challenging datasets show that the CMGFNet outperforms other state-of-the-art models. The efficacy of all significant elements is also confirmed by the extensive ablation study

Precise 3D extraction of building roofs by fusion of UAV-based thermal and visible images

Thermography is an efficient way of detecting the thermal problems of the roof as a major part of a building’s energy dissipation. Thermal images have a low spatial resolution, making it a challenge to produce a three-dimensional thermal model using aerial images. This paper proposes a combination of thermal and visible point clouds to generate a higher-resolution thermal point cloud from roofs of buildings. For this purpose, after obtaining the internal orientation parameters through camera calibration, visible and thermal point clouds were generated and then registered to each other using ground control points. The roofs of buildings were then extracted from the visible point cloud in four steps. First ground points were removed using cloth simulation filter (CSF), and then vegetation points were eliminated by applying entropy and red-green-blue vegetation index (RGBVI). Geometric features and a segmentation step were considered to filter roofs from other parts. Finally, by combining visible and thermal point clouds, the generated point had a high spatial resolution along with thermal information. In the achieved results, the thermal camera calibration was performed with an accuracy of 0.31 pixels, and the thermal and visible point clouds were registered with an absolute distance of < 0.3 m. The experimental results showed an accuracy of 18 cm for automated extraction of building roofs and 0.6 pixel for production of a high-resolution thermal point cloud, which was five times the density of the primary thermal point cloud and free from distortions

Potential evaluation of visible-thermal UAV image fusion for individual tree detection based on convolutional neural network

Unmanned aerial vehicles (UAVs) outfitted with thermal and visible sensors are already a popular platform in precision agriculture thanks to recent advances in remote sensing. Many researchers have studied integrating data from sensors with different spectral characteristics to achieve higher-level properties and, consequently, detect the trees accurately. In this research, visible and thermal images, as well as normalized digital surface models resulting from UAVs with high spatial resolution, are employed to accurately extract trees from two studied urban areas with complex backgrounds. In the thermal image, trees can be detected in hidden areas based on their brightness temperature difference compared to other features. In contrast, the visible image has a higher spatial resolution, and fusing this data with thermal images can resolve the complexity of the problem. In the proposed method, first, a deep learning network based on visible-thermal data is evaluated in terms of detecting trees with various data approaches. These evaluations include comparison tests on four types of data input to the convolutional network of the visible images, thermal images, fusing visible-thermal images, and also fusing visible-thermal- normalized digital surface model images. Results of evaluation parameters indicate maximum precision in the fourth approach (intersection-over-union = 91.72, F-score = 95.67). Then, the output binary map with the highest accuracy approach and Canny edge detection operator is utilized to accurately identify tree boundaries, count, and estimate the area and diameter of the tree canopy. Finally, the findings revealed the root mean square error (RMSE) first and second areas are 0.21 m2, 0.08 m and 0.24 m2, 0.11 m respectively for the area and diameter of the tree crown

Detection and recognition of drones based on a deep convolutional neural network using visible imagery

Drones are becoming increasingly popular not only for recreational purposes but also in a variety of applications in engineering, disaster management, logistics, securing airports, and others. In addition to their useful applications, an alarming concern regarding physical infrastructure security, safety, and surveillance at airports has arisen due to the potential of their use in malicious activities. In recent years, there have been many reports of the unauthorized use of various types of drones at airports and the disruption of airline operations. To address this problem, this study proposes a novel deep learning-based method for the efficient detection and recognition of two types of drones and birds. Evaluation of the proposed approach with the prepared image dataset demonstrates better efficiency compared to existing detection systems in the literature. Furthermore, drones are often confused with birds because of their physical and behavioral similarity. The proposed method is not only able to detect the presence or absence of drones in an area but also to recognize and distinguish between two types of drones, as well as distinguish them from birds. The dataset used in this work to train the network consists of 10,000 visible images containing two types of drones as multirotors, helicopters, and also birds. The proposed deep learning method can directly detect and recognize two types of drones and distinguish them from birds with an accuracy of 83%, mAP of 84%, and IoU of 81%. The values of average recall, average accuracy, and average F1-score were also reported as 84%, 83%, and 83%, respectively, in three classes